A Practical Video Database Based on Language and Image Analysis

We integrated a practical digital video database system based on language and image analysis with components from digital video processing, still image search, information retrieval, dosed captioning processing. The attempt is to utilize the multiple modalities of information in video and implement data fusion among the multiple modalities. Keyframes are extracted to represent shots based on video content. Logical structure of video is discovered and represented as Scene Transition Diagram. Storyboard and STG, and catalog information provide means to browse video content non-linearly. Text information from dosed captioning is extracted and associated with shots. Text information search and image similarity search are provided to get direct access to video shots. We integrated a practical digital video database system based on language and image analysis with components from digital video processing, still image search, information retrieval, closed captioning processing, and Intemet and Web page programming. The research and development project currently has digitized video programs of total 13 hours and 50 minutes long, among which one hour and 5 minutes are stored at Advance Inc. and 12.75 hours are stored at Princeton University. These MPEG video stories take up about 8.4 GB disk space. The contents are focused on two areas: US military operation in Bosnia and US presidential politics. 1. Multiple Modality from Video Information One key feature of our research is data fusion (i.e., making use of all types of existing information which coexists in video information). What is needed is a system that can utilize multiple tracks of information embedded in digital video in order to obtain retrieval terms that characterize the situation shown in the video. This should include audio, video, closed captions, caption, sound, music, special effects, catalog information, and transcripts. Audio, text, sound track, and image analysis should work hand-in-hand to disambiguate the terms, with imagery identifying speakers or objects, with language analysis providing keys needed for image analysis, and the sound track (i.e., music and special effects) assisting further interpretation of the contents. Text from Speech Recognition Extracting transcripts from audio track through speech recognition was the first task examined. Our intention is to study the application of existing speech recognition algorithms for the extraction of words from the audio track. The development of new speech recognition algorithms, although not the goal of our research, was examined for the purpose of extracting keywords as part of the data fusion effort. The intent of this research is to discover an appropriate Speech Recognition system capable of performing dictation and transformation of human speech from the audio portion of digitized video data into text for use in text inquiries. A Speech Recognition engine must be capable of the following: Speaker-independent: requirement of preparing various individuals in a video must be a bare minimum. The system should be adaptive in order to increase recognition accuracy. ̄ Continuous speech recognition: system must be able to perform recognition of continuous natural human speech. Discrete speech recognition requires short silences to separate individual words, thereby performing poorly on ordinary human speech. Speaker and vocabulary adaptation is desirous. Unrestricted dictation: restriction in term of noiselevel, domain-specifics, or vocabulary-size are not acceptable in the application. Many commercial packages for speech recognition were reviewed including ones in the research effort. Special attention was paid to packages capable of reading from MPEG files in order to recognize speeches in view of our plan to use MPEG as the digital video standard and the fact that MPEG files for video and audio can be generated at digitization time. None of the reviewed packages including IBM VoiceType Dictation, Microsoft Whisper, PowerSecretary from Articulate Systems, Inc., AT&T WATSON Advanced Speech Applications Platform, DragonDictate Dragon System, Philips SpeechMagic can 127 From: AAAI Technical Report SS-97-03. Compilation copyright © 1997, AAAI (www.aaai.org). All rights reserved.